Background Estimation and Removal Based on Range and Color

Background estimation and removal based on the joint use of range and color data produces superior results than can be achieved with either data source alone. This is increasingly relevant as inexpensive, real-time, passive range systems become more accessible through novel hardware and increased CPU processing speeds. Range is a powerful signal for segmentation which is largely independent of color, and hence not effected by the classic color segmentation problems of shadows and objects with color similar to the background. However, range alone is also not sufficient for the good segmentation: depth measurements are rarely available at all pixels in the scene, and foreground objects may be indistinguishable in depth when they are close to the background. Color segmentation is complementary in these cases. Surprisingly, little work has been done to date on joint range and color segmentation. We describe and demonstrate a background estimation method based on a multidimensional (range and color) clustering at each image pixel. Segmentation of the foreground in a given frame is performed via comparison with background statistics in range and normalized color. Important implementation issues such as treatment of shadows and low confidence measurements are discussed in detail. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (Fort Collins, CO), June 1999. c 1999 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in orther works must be obtained from the IEEE. 1 Motivation Separating dynamic objects, such as people, from a relatively static background scene is a very important preprocessing step in many computer vision applications. Accurate and efficient background removal is critical for interactive games[7], person detection and tracking[1, 4], and graphical special effects. One of the most common approaches to this problem is color or greyscale background subtraction. Typical problems with this technique include foreground objects with some of the same colors as the background (produce holes in the computed foreground), and shadows or other variable lighting conditions (cause inclusion of background elements in the computed foreground). In this paper we present a passive method for background estimation and removal based on the joint use of range and color which produces superior results than can be achieved with either data source alone. This approach is now practical for general applications as inexpensive real-time passive range data is becoming more accessible through novel hardware[10] and increased CPU processing speeds. The joint use of color and range produces cleaner segmentation of the foreground scene in comparison to the commonly used color-based background subtraction or rangebased segmentation. Background subtraction based on color or intensity is a commonly used technique to quickly identify foreground elements. In current systems [3, 4, 11] performance is improved by using statistical models to represent the background (e.g single or multiple Gaussians at each pixel), as well as updating these models over time to account for slow changes. There are two classic problems with this approach. Clearly, if regions of the foreground contain similar colors as the background, they can be erroneously removed. Also, shadows cast on the background can be erroneously selected as foreground. This problem can be minimized by computing differences in a color space (hue, log color opponent, intensity normalized RGB[11]) which is less sensitive to intensity change. However, it is difficult to optimize a single match criterion such that it allows most shadowed pixels to match their normal background color and does not allow regions of the true foreground to match background pixels with similar hue. Figure 1 shows an example of color based segmentation failure. Range has also been used for background removal[2, 5, 6]. The main issue in this approach is that depth computation via stereo, which relies on finding correspondences between two images, does not produce valid results in low contrast regions or in regions which can not be seen in both views. In our stereo implementation (described in section 2.1), these low confidence cases are detected and marked with a special value we will refer to as invalid . It Figure 1. Color background subtraction has difficulty when portions of the foreground include the same colors as the background. Top left shows color background model, top right shows color image from scene. The bottom image shows segmentation results from comparison of these images. The range background model and image are also shown for reference, although they are not used in this segmentation. is rare that all pixels in the scene will have valid range on which to base a segmentation decision. It is also difficult to use range data to segment foreground objects which are at approximately the same distance as the background. Figure 2 shows an example of range based segmentation failure. We present a scheme which takes advantage of the strengths of each data source for background modeling and segmentation. Background estimation is based on a multidimensional (range and color) mixture of Gaussians which can be performed for sequences containing substantial foreground elements. Segmentation of the foreground is performed via background comparison in range and normalized color. For optimal performance, we find we must explicitly take into account low confidence values in range and color, as well as shadow conditions. The background estimation is described in section 2, followed by the segmentation method in section 3. Figure 2. Middle images show range background model and new scene image. Stereo computation can not produce valid range estimates in areas which have very low texture (e.g. saturated regions) or which are occluded in one view. Invalid range values are shown in white. Depth based segmentation, shown in bottom image, will fail in regions of the foreground which are undefined in depth. Top row shows color backgroundmodel and scene image for reference, although they are not use in segmentation. Color of the foreground is overlayed on the segmentation results for easier interpretation. 2 Background Estimation In basic terms, we define the background as the stationary portion of a scene. Many applications simply require that there be introductory frames in the sequence which contain only background elements. If pure background frames are available, pixel-wise statistics in color and depth can be computed directly. The more difficult case is computing the background model in sequences which always contain foreground elements. We model each pixel as an independent statistical process. We record the (R,G,B,Z) observations at each pixel over a sequence of frames in a multidimensional histogram. We then use a clustering method to fit the data with an approximation of a mixture of Gaussians. For ease of computation, we assume a covariance matrix of the form = 2I . At each pixel one of the clusters is selected as the background process. The others are considered to be caused by